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ABSTRACT 



For centuries, cameras were designed to closely mimic the human visual system. Over the last decade, due to the 
rapid increase in computational power, researchers in the computer vision, graphics and optics community have begun to 
focus their attention on new types of imaging systems that utilize computations as an integral part of the imaging process. 
Computational cameras uses a combination of optics and software to produce images by optically encoding information 
that later decoded using signal processing. Over the last decade, a number of Computational Imaging (CI) systems have 
been proposed for tasks such as motion deblurring, defocus deblurring and multispectral imaging. These techniques 
increase the amount of light reaching the sensor via multiplexing and then undo the deleterious effects of multiplexing by 
appropriate reconstruction algorithms. While computational techniques can be used to increase optical efficiency, 
this comes at a cost. The cost incurred is noise amplification caused by the decoding process. Thus, to measure the real 
utility of a computational approach, we must weigh the benefit of increased optical efficiency against the cost of amplified 
noise. A complete treatment must take into account an accurate noise model. In some cases, the benefit may not outweigh 
the cost, and thus a computational approach has no value. This paper concludes with a discussion on these scenarios. 

KEYWORDS: Computational Imaging, Computational Photography, Defocus De blurring, Motion Deblurring, 
Multiplexing 

INTRODUCTION 

Digital photography replaced the photographic film by an electronic sensor. Despite of its practical implications, 
this phenomenon limited itself to replace the chemical process of film development by an electronic gathering of photons. 
Current technology, however, allows us to expand the capabilities of photography with new and exciting possibilities. 
The most strategic way of achieving this is by combining information from multiple images acquired with a single camera 
by slightly different parameter values, such as position/orientation, exposure time, and aperture size. Such a strategy can be 
used to construct panoramas, and high-dynamic-range images. In the last decade, computational imaging has emerged as a 
popular field of research. When compared with conventional cameras, a wide variety of computational cameras have been 
observed to encode better visual information in the captured images. A computational camera uses a combination of optics 
and software to produce images that cannot be taken with traditional cameras. Moreover, computational photography 
techniques overcome limitations of traditional image sensor such as dynamic range and noise. Many computational 
imaging techniques have been proposed that process image stacks acquired using different exposure, aperture or gain 
setting. 

A number of CI techniques have been introduced to improve image quality by increasing light throughput. 
These techniques use optical coding to measure a stronger signal level. The optically coded signal is then decoded. 
However, the performance of these techniques is limited by the decoding step, which amplifies noise. While it is well 
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understood that optical coding can increase performance at low light levels, little is known about the quantitative 
performance advantage of computational imaging in general settings. [1] 

Performance improvement achieved via multiplexing has received a fair amount of attention in the literature [1], 
[2], [3], [4], [5]. It is well understood that multiplexing gives the greatest advantage at low light levels 
(where signal-independent read noise dominates), but this advantage diminishes with increasing light 
(where signal dependent photon noise dominates). 

As per the convention adopted by Cossairt et al. [1], we follow a line of research whose goal is to relate maximum 
performance of CI techniques to practical considerations (e.g. illumination conditions and sensor characteristics) [1], in this 
work. We have considered the definition by him as conventional camera is an impulse imaging system which measures the 
desired signal directly. We then compare the CI performance against the impulse imaging system. Defocus and motion blur 
are the two most common issues in any imaging system. This type of blur can be position dependent when objects in the 
scene span either a range of depths or velocities. Various techniques have been devised to encode blur so as to make it either 
well-conditioned or position-independent (shift-invariant), or both. In this paper, we pay special attention to the problems of 
defocus and motion blur. For defocus deblurring, CI systems encode defocus blur using attenuation masks [7], [8], [6]. 
The impulse imaging counterpart is a narrow aperture image with no defocus blur. For motion deblurring, CI systems 
encode motion blur using a fluttered shutter [10] or camera motion [9]. The impulse imaging counterpart is an image with 
short exposure time and no motion blur. 

COMPUTATIONAL IMAGING 

Computational photography combines plentiful computing, digital sensors, modern optics, actuators, and smart 
lights to escape the limitations of traditional cameras, enables novel imaging applications and simplifies many computer 
vision tasks. The CI systems can be broadly classified into two categories: One designed to add a new functionality or other 
to increase performance relative to a conventional imaging system. A light field camera is an example of the former, 
which is capable to refocus or change perspective even after images are captured - a functionality impossible to achieve 
with a conventional camera. Some of the example of the latter type are Motion deblurring [10], [9], illumination 
multiplexing [4], [5] and are the main focus of this paper. These systems use optical coding (multiplexing) to increase light 
throughput, which increases the SNR of captured images. The desired signal is then recovered computationally via signal 
processing. The quality of recovered images jointly depends on the conditioning of the optical coding and the increased light 
throughput. A poor choice of multiplexing will reduce image quality. 

Computational imaging systems have gained huge popularity due to two main reasons. The first is that they offer 
increased functionality relative to a conventional imaging system by translating the ability of capturing new types of visual 
information. There are abundant functions that computational cameras enable which are not accessible via conventional 
cameras - including digital refocusing, depth estimation, digital refocusing, digital perspective adjustment, multispectral 
capture, motion blur removal and defocus blur removal. The second reason is that they can offer a performance advantage 
over a conventional imaging system, which translates directly into greater fidelity in measurement and robustness to noise. 
When computational cameras increase optical efficiency, they increase the strength of captured signals and at often times 
this can lead to an increase in performance. 
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Image Formation Model 

In this paper we assume a linear image formation model that can be represented as given by Cossairt et al. [1], 

g = **f + n 

(i) 

Where g is the measurements vector of size N. f is the vector of unknown signal we want to capture, H is N x N 
measurement matrix. For CI techniques that take coded measurements by masking (attenuating) light, the entries of H are 
between and 1. For CI techniques that measure the signal without masking light, either by moving the sensor during 
capture, using additional refractive elements or moving the camera, the entries of H are not bounded. For impulse imaging, 
H = I, and the camera measures the signal f directly, and r| is observation noise. 

Noise Model 

We use an affine noise model to describe the combined effects of signal-independent read noise, and signal 
dependent photon noise. The photon noise can be approximated by a Gaussian with variance equal to the measured signal 
level / (in photons). Let the variance of the read noise be o> . The total noise variance is [1]: 

fT 2 = J -+- €7?. 

(2) 

Functionality, Resolution and Blur 

We have broadly defined functionality as the ability to flexibly sample the radiometric and geometric properties of 
a scene. An important aspect of sampling is the resolution that we can sample at. For conventional imaging, choosing the 
sampling resolution amounts to choosing the size of the support of the sampling basis. We typically want to sample at as 
high resolution as possible, which would indicate that we want to choose small support. However, the choice of sampling 
resolution has a large impact on the amount of image blur exhibited by the imaging system. In this paper, we pay attention to 
the problems of defocus and motion blur. 

Cameras exhibit motion blur when objects move during exposure so that points in the image are blurred along the 
direction of motion. A scene may consist of multiple objects moving at different speeds and directions. Temporal shuttering 
has been used to remove motion blur from images [10]. Methods have also been proposed that create motion -invariant blur 
that can be removed without prior knowledge of object speed [l].In contrast, impulse imaging avoids motion blur by simply 
capturing images with a short exposure 

Cameras exhibit defocus blur when objects are located at depths other than the focal plane of the camera. 
The further objects are from the focal plane, the greater the amount of defocus blur will be. The problem of defocus blur 
arises due to the finite size of the camera aperture. Pinhole cameras exhibit no defocus blur, while larger aperture sizes 
introduce greater amounts of defocus blur. We can thus always remove defocus blur by stopping down the aperture. We do 
so however, at the cost of reducing the amount of light captured by the sensor. An alternative to stopping down the aperture 
is to use an EDOF camera to produce a depth-invariant blur. Because the blur is depth invariant, it can be removed via 
deconvolution. However, decon volution amplifies the noise in captured EDOF images. 
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PERFORMANCE LIMITS FOR CI 

Performance offered by computational cameras should always be evaluated against performance of conventional 
cameras. Computational cameras allow blur to be removed, and at the same time maintain high optical efficiency. 
However, we can always remove blur by using a conventional camera that is less optical efficiency (i.e. we can reduce 
exposure time for motion blur, or reduce aperture size for defocus blur). Therefore, when we evaluating the performance of a 
computational camera, we need to compare against the performance of a conventional camera. 

As an example, consider the problem of defocus blur. Both the pinhole camera and the coded aperture camera can 
produce an image that is free of blur, however, the coded aperture camera captures an image with much greater optical 
efficiency. We have a vague sense that greater optical efficiency is desirable because it increases the signal strength of 
captured images, but we still haven't determined concretely which technique produces better performance: the pinhole or 
coded aperture camera. There are two determining factors in evaluating performance: the conditioning of the transfer matrix 
(H) and the noise model. When we code the aperture, we increase the conditioning of the transfer matrix so that blur can be 
removed without sacrificing optical efficiency. However, depending on the noise model, an increase in efficiency may 
actually increase the noise level as well as increasing the signal strength. So we need to be more specific about the noise 
model before we can make any concrete statements about the performance of computational cameras. 

When to Use Computational Imaging 

In order to determine when CI is advantageous, use following expression of Cossairt [1] for the signal level / which 
depends on two parameters i.e. scene and sensor dependent parameters, as mentioned below: 

J = 1 O 15 (/V^) -2 * *src R q A 2 

Scene Sensor 
Dependent Dependent 

Where F/# is the ratio of focal length to aperture size of the lens, t is the exposure time, Isrc is the incident 
illuminance given in lux, R is the average reflectivity of the scene, q is the quantum efficiency of the sensor, and A is the 
pixel size in meters. Figure 1 shows an example plot of performance for CI techniques relative to impulse imaging. We can 
observe that computational imaging techniques do not provide a significant performance advantage when imaging with 
illumination that is brighter than typical daylight. 
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Figure 1 (O Cossairt [1]) Performance of computational imaging for naturally occurring lighting conditions. 
We show that CI techniques (solid curve) give a negligible performance gain over conventional (impulse) imaging 
(dotted line) if the illumination level is higher than that of a typical living room. This example plot for spectral, light field, 
and illumination multiplexing systems for the following scene and sensor characteristics: average scene reflectivity is 0.5, 
exposure time is 20ms, aperture setting is F/2.1, pixel size is 1 um, quantum efficiency is 0. 5, and read noise standard 
deviation is 4e-. 

Computational versus Impulse Imaging 

For many CI techniques, there is a corresponding conventional imaging technique that can measure the signal 
directly without the need for any decoding. For example, a stopped down aperture can be used to avoid defocus blur and a 
shorter exposure can be used to eliminate motion blur. In this paper, we refer class of conventional imaging methods as 
impulse imaging. The term impulse is meant to convey the small amount of light captured by these methods. Figure 2 gives 
some example comparisons between CI techniques and their impulse imaging counterparts. For defocus deblurring, 
CI systems encode defocus blur using attenuation masks, refractive masks, or motion. The impulse imaging counterpart is a 
narrow aperture image with no defocus blur. For motion deblurring, CI systems encode motion blur using a fluttered shutter 
or camera motion. The impulse imaging counterpart is an image with short exposure time and no motion blur. 
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Figure 2 

Figure 2 (Cossairt [1]) Computational versus Impulse Imaging. (Left) CI techniques discussed can be modelled 
using a linear image formation model. This includes defocus deblurring, motion deblurring, spectral multiplexing, and many 
others. In order to recover the desired image, these techniques require an additional decoding step, which amplifies noise. 
(Right) Impulse imaging techniques measure the signal directly without requiring any decoding. A stopped down aperture 
can be used to avoid defocus blur, a shorter exposure can be used to avoid motion blur. 

CONCLUSIONS 

There are two main reasons why we use computational imaging systems. The first is that they offer increased 
functionality relative to a conventional imaging system. The increase in functionality translates to the ability to capture new 
types of visual information. There is a whole plethora of functions that computational cameras enable which are not 
accessible via conventional cameras - including depth estimation, digital refocusing, digital perspective adjustment, 
multispectral capture, motion blur removal, and defocus blur removal. However, new functionality is not the only reason we 
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use computational cameras. The second reason is that they can offer a performance advantage relative to a conventional 
imaging system, which translates directly into greater fidelity in measurement and robustness to noise. When computational 
cameras increase optical efficiency, they increase the strength of captured signals, and often times this can lead to an 
increase in performance. 

Since decoding involve in computational imaging amplifies noise its performance should always compared with 
impulse images where decoding is not required .Also, we studied that computational imaging techniques do not provide a 
significant performance advantage when imaging with illumination that is brighter than typical daylight. 
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